Papers by Jonas F. Lotz
Multilingual Pretraining for Pixel Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | PIXEL-M4 model pretrains on four visually and linguistically diverse languages . previous work on pixel-based language models focused on monolingual pretraining on English data . |
| Approach: | They propose a pixel-based language model that is pretrained on four visually diverse languages. |
| Outcome: | The proposed model outperforms an English-only counterpart on non-Latin scripts on semantic and syntactic tasks. |
Beyond Text Compression: Evaluating Tokenizers Across Scales (2025.acl-long)
Copied to clipboard
| Challenge: | Language models rely on tokenizers to convert text into machine-interpretable tokens, which shape the statistical patterns that language models learn to estimate. |
| Approach: | They propose to use Zipf's law to measure tokenizer performance by combining several metrics to capture multiple aspects of tokenizer behavior. |
| Outcome: | The proposed metrics correlate more strongly with downstream performance than text compression when modeling unseen languages. |